Path-based ranking of unvisited web pages

ABSTRACT

Path-based ranking of unvisited Web pages for WWW crawling is provided, via identifying all the paths beginning with a “seed” URL and leading to visited relevant web pages as “good-path set”, and for each unvisited web page, identifying the paths beginning from the “seed” URL leading to it as “partial-path set”; classifying all the visited web pages and labeling each web Page with the labels of a class or classes it belongs to; training a statistic model for generalizing the common patterns among all ones of “good-path set”; and evaluating the “partial-path set” with the statistic model and ranking the unvisited web pages with the evaluation results.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 10/358,941 filed on Feb. 5, 2003, issued as U.S.Pat. No. 7,424,484, which claims priority from Chinese Application No.02 1 03529.6, filed on Feb. 5, 2002. The entire contents of theseapplications are incorporated herein by reference.

FIELD OF THE INVENTION

The invention relates to network information searching, and inparticular to a method and system for path-based ranking of unvisitedWeb pages for WWW crawling.

BACKGROUND OF THE INVENTION

Existing general purpose search engines provide valuable assistance tousers in locating the information relevant to their needs on the WorldWide Web. However, they are unsatisfactory when users try to findin-time information for a narrow query in a specific domain. It isestimated that only 30-40% of the Web pages are collected and put intothe search engine index by the largest crawls, and the completerefreshing takes several weeks to a month, thus much of the up-to-dateinformation is out of the search scope. Another drawback of generalsearch engines is: it makes a loss of much information in the Web pages,while it enables fast searching to build a content index.

“Focused crawling” is recognized as a promising solution to satisfy theabove search requirements. It can collect the useful information with avery limited resource. For example, users are already using PC based“Focused crawling” implementations. It can also exploit plentifulinformation hidden in the original web pages as well as the web topologyto make more accurate judgment about the relevance.

“Focused crawling” is an intelligent way to crawl the Word Wide Web andcollects only the web pages relevant to a specific information need. Inparticular, the “crawler” begins from a “seed” web page andintelligently visits other web pages following the links in the “seed”web page. And then, the “crawler” follows the links in visited webpages. As this process goes on, the number of possible links or theirtarget web pages increases in an explosion way. The challenge is how tomake the “crawler” visit as many relevant web pages as possible giventhat the number of total visited web pages is limited by time, networkbandwidth and other resource restrictions. In the implementation of a“crawler”, the challenge is boiled down to make decisions on which amongthe unvisited web pages should be visited in priority.

Known ranking methods only took advantage of “local” information in theweb page. The “local” information in a web page includes the number ofin-links, keywords and their positions, etc. However, the paths docontain valuable information for “focused crawling”. For example,usually you can find research projects on artificial intelligence with apath like “University homepage”-“Academies”-“College (school,department) of Computer Science”-“Research Areas”. Actually, peopleshare a similar knowledge structure and they cope the structure whenbuilding the web site thus make similar patterns. This invention meetsthe challenge with a novel way to rank candidate web pages (representedby the URLs pointing them) by path-based ranking the pages so thatoptimal crawling is made.

SUMMARY OF THE INVENTION

In accordance with at least one presently preferred embodiment of thepresent invention, there is broadly contemplated a system whichpath-based ranks unvisited Web pages. The system can be built as astand-alone marketable software product, an addition to another system,or as a service such as a web-based service.

In summary, one aspect of the invention provides a system for path-basedranking of unvisited Web pages for WWW crawler, comprising: a storerwhich stores all paths leading to visited web pages containing contentrelevant to a search as “good-path set” and all paths leading tounvisited web pages as “partial-path set”; a classifier and a labelerwhich classifies all the visited web pages and labels each web page withthe labels of a class or classes it belongs to; a statistic modelerwhich trains a statistic model for generalizing the common patternsamong all ones of “good-path set”; and an evaluator which evaluates the“partial-path set” with the statistic model to determine a similaritybetween the partial-path set and all ones of good-path set and ranks theunvisited web pages with the evaluation results; wherein the highestranked unvisited web pages are those most likely to contain contentrelevant to the search.

A further additional aspect of the present invention provides a programstorage device readable by machine, tangibly embodying a program ofinstructions executable by the machine to perform method steps forpath-based ranking of unvisited Web pages for WWW crawling, said methodcomprising the steps of: identifying all the paths beginning with a“seed” URL and leading to visited web pages containing content relevantto a search as “good-path set”, and for each unvisited web page,identifying the paths beginning from the “seed” URL leading to it as“partial-path set”; classifying all the visited web pages and labelingeach web page with the labels of a class or classes it belongs to;training a statistic model for generalizing the common patterns amongall ones of good-path set; and evaluating the partial-path sets with thestatistic model to determine a similarity between the partial-path setsand all ones of good-path set and ranking the unvisited web pages withthe evaluation results; whereby the highest ranked unvisited web pagesare those most likely to contain content relevant to the search.

For a better understanding of the present invention, together with otherand further features and advantages thereof, reference is made to thefollowing description, taken in conjunction with the accompanyingdrawings, and the scope of the invention will be pointed out in theappended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a conventional crawling process;

FIG. 2 shows a method for path-based ranking of unvisited Web pagesaccording to a preferred embodiment of the invention;

FIG. 3 shows a system for path-based ranking of unvisited Web pagesaccording to a preferred embodiment of the invention;

FIGS. 4-6 show an example of the crawling path.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to FIG. 1, the crawling process goes as followings:

(101) Adding “seeds” to the unvisited list;

(102) Choosing one or several URLs from the unvisited list, downloadingpointed web pages into local storage and adding the URLs to the visitedlist;

(103) Checking and marking the web page as whether they are relevant tothe information needs;

(104) Extracting URLs embedded in the downloaded web pages; if they areneither in visited list nor in unvisited list, adding them to unvisitedlist;

(105) If stop condition is met, stop the process and returned to the webpages marked relevant, otherwise back to (102).

The invention concerns step (102) where decisions are made on whichlinks should be followed. The method and system for path-based rankingof unvisited Web pages according to the invention improves theefficiency of the total crawling.

FIG. 2 shows a method for path-based ranking of unvisited Web pagesaccording to a preferred embodiment of the invention. As shown in FIG.2, according to one preferred embodiment, the path-based ranking ofunvisited Web pages process is as follows:

(201) Representing linkage structure among all the visited web pageswith a directed graph, which consists of web pages as nodes and linksbetween pages as edges;

(202) Identifying all the paths beginning with the “seed” and leading tothe visited relevant web pages (denoted as the “good-path set”);

(203) For each unvisited web page, identifying the paths leading to it(denoted as the “partial-path set” corresponded to the unvisited webpage) beginning from the “seed”.

(204) Adjusting the good-path set through a Human-Computer Interactionstage;

(205) Classifying all the visited web pages and labeling each web pagewith the labels of a class or classes it belongs to;

(206) Training a statistic model for generalizing the common patternsamong all the ones of good-path set;

(207) For each path in partial-path sets, calculating the similaritymeasure against the “good-path set” with the statistic model;

(208) Evaluating every partial-path set with the similarity measures ofits members;

(209) Ranking the unvisited web pages using the evaluating results forits corresponding partial-path set.

The ranking process will now be described by reference to FIGS. 4-6. Thecrawling path shown in FIG. 4 will be obtained after executing thefollowing steps:

Step 201: drawing out the linkage structure in a directed graph; steps202 and 203: identifying elements for the good-path set and thepartial-path set; step 204: adjusting the good-path set; step 205:classifying and labeling web pages.

FIG. 4 illustrates the crawling paths. The paths are with a left toright direction. The note denotes a web page with a class label (thereference number in the FIG. 4). The following description demonstratesa process by which it could label a web page for modeling and ranking.

At step 201, a feature vector that describes the web page is extractedfor each web page from a set of web pages as training samples. Thefeature vectors may consist of the TF-IDF (Term Frequency InverseDocument Frequency) of:

(1) words in the text elements of a web page, or

(2) words in the anchor text element of a web page.

The TF-IDF score e(w) of a word w is defined as the following,

${e(w)} = {\frac{f^{d}(w)}{f_{\max}^{d}}\log\frac{N}{f(w)}}$where f_((w)) ^(d) is the number of occurrences of w in a web pagedocument d. f_(max) ^(d) is the maximum umber of occurrences of a wordin the document. N is the number of documents in the corpus, and f(w) isthe number documents in the corpus where the word occurs at least once.

At step 202, using the k—means algorithm or other VQ (Vector Quantify)techniques to train a cluster and its codebook.

At step 203, the ready cluster with its codebook could label any webdocument. It compares the feature vector of a candidate web documentwith all the feature vectors recorded in its codebook and classifieswhich class the vector should belong to. Then it assigns thecorresponding label of the class to the candidate web page.

At step 204, after initial labeling, adjacent nodes may be merged intoone node if they share the same label.

At step 205, each crawling path can be treated as a realization of arandom process in mathematics. There exist many mathematics solutionsfor modeling the underlying random process based on the observedrealizations.

At step 206, a random process model (also a statistic model) extractsthe common patterns among the good paths. It also suggests a probabilitymeasure on any new path. As a fact, the good paths may be described bythe models in the following statistic form,P(L _(i)|(L _(i-1) ,L _(i-2) , . . . , L ₀)), i≧0  (1)where L_(i) denotes a web page's label and the web page is the ith onein a path whose length is defined as i+1. (L_(i-1),L_(i-2), . . . , L₀)denotes a certain path having the length of i and whose nth web page iswith the label of L_(n), 0≦n≦i. Such a model can be trained out by theobserved samples (the ones in a good path set).

The estimation of formula (1) can be calculated in a very straight way.An example can be like this,

${P\left( {L_{i}❘\left( {L_{i - 1},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)} \right)} = \frac{\left. {f\left( {L_{i - 1},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)} \right)}{f\left( {L_{i - 1},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)}$Where, f(L_(i-1),L_(i-2), . . . , L₀) is occurrence number of such anevent that a path (L_(i-1),L_(i-2), . . . , L₀) lead to a web pagelabeled as L_(i), and f(L_(i-1),L_(i-2), . . . , L₀) denotes theoccurrence number of such a path (L_(i-1), L_(i-2), . . . , L₀) in thetraining set.

At step 207, for ranking any a new path, it could define the measurementof any a path as Mea (Path_(x)) which suggests the similarity betweenthe path and the good path set. The measurement is a function like theseformulas,Mea(Path_(x))=P(Path_(x))  (2)orMea(Path_(x))=_(N) ^(Σ) C(N)*_(Path) _(y) _(N) ^(Σ) P(T,Path_(y)^(N),Path_(x))  (3)where T denotes a target web page. The Path_(y) ^(N) is a path in lengthN. C(N) is a cost weight associated with the distance between thecurrent page and the target.

The difference between formula (2) and formula (3) is that: Themeasurement in formula (2) only depends on history of crawling to thecurrent page. The measurement in formula (3) also considers the paths tobe followed before reaching the target.

The probability

in the formulas could be further expanded as the following,P(Path_(x))=P(L _(i) ,L _(i-1) ,L _(i-2) , . . . , L ₀)  (4)

The probabilities are indeed calculable under the model,

$\begin{matrix}{{P\left( {L_{i},L_{i - 1},L_{i - 2},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)} = {{P\left( {L_{i}❘\left( {L_{i - 1},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)} \right)}*\mspace{101mu}(5)}} \\{P\left( {L_{i - 1},L_{i - 2},\ldots,L_{0}} \right)} \\{= {{P\left( {L_{i}❘\left( {L_{i - 1},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)} \right)}*P}} \\{\left. \left( {L_{i - 1},L_{i - 2},L_{i - 3},\ldots\mspace{14mu},L_{0}} \right) \right)*} \\{P\left( {L_{i - 2},L_{i - 3},\ldots\mspace{14mu},L_{0}} \right)} \\{= {{P\left( {L_{i}❘\left( {L_{i - 1},L_{i - 2},\ldots\mspace{14mu},L_{0}} \right)} \right)}*}} \\{\left. {P\left( {{L_{i - 1}❘L_{i - 2}},L_{i - 3},\ldots\mspace{14mu},L_{0}} \right)} \right)*} \\{{P\left( {L_{i - 2}❘\left( {L_{i - 3},L_{i - 4},\ldots\mspace{14mu},L_{0}} \right)} \right)}*\ldots*{P\left( L_{0} \right)}}\end{matrix}$

In an implementation of this invention, other limitations andassumptions could be introduced to the properties of underlying patternswhich will simplify the applications of the method in mathematics. Foran example, one could assume that all the good paths be realizations ofa random process with Markov Properties.

Various accumulative set functions can be applied at step 208, here twoexamples are given to evaluate a partial-path set x.

$\begin{matrix}{{J(X)} = {\underset{{Path}_{x} \in X}{Max}\left( {{Mea}\left( {Path}_{x} \right)} \right)}} \\{{J(X)} = {\underset{{Path}_{x} \in X}{Max}\left( {{Mea}\left( {Path}_{x} \right)} \right)}}\end{matrix}$orwith the results from above steps, step 209 is simply a numeric sortingtask.

This invention allows to obtain more efficient models through modifyingthe training sample paths (in step 204). It gives the following examplemodification methods:

(1) A user may choose URLs in his “My favorite” list as targets,back-crawl all the paths leading to the targets with “connectivityserver” such as Google and Altavista and append the paths into thelinkage structure;

(2) A HCI engine may visualize the linkage structure and the attributesof nodes, such as the title, URL etc, to the user. Based on the suppliedinformation, the user may delete a path from the good-path set;

(3) The invention also allows the ability of self-learning throughupdating the model in time. As a result, the longer does the crawleract, the better does it work.

FIGS. 5 and 6 show the crawling example. FIG. 5 gives examples ofpossible good paths. The models derived from the samples can look likethese,

P(1)=0.207, P(2)=0.345, P(3)=0.276, P(4)=0.103, P(5)=0.069

P(1|1)=0.333, P(2|1)=0.167, P(5|1)=0.167, P(4|1)=0.333,

P(2|2)=0.300, P(3|2)=0.500, P(T|2)=0.200,

P(2|3)=0.250, P(3|3)=0.250, P(4|3)=0.125, P(T|3)=0.375,

P(2|4)=0.667, P(3|4)=0.333,

P(2|5)=0.500, P(1|5)=0.500,

P(i|j)=0.001 and P(i)=0.001 for others

Then the model can be used to measure the similarity between a partialpath and the good path. FIG. 6 shows some partial paths. The statisticmodel is employed to measure the similarity of these partial paths.

Log(Mea(Path1))=−7.337

Log(Mea(Path2))=−8.027

Log(Mea(Path3))=−2.940

Log(Mea(Path4))=−15.426

Log(Mea(Path5))=−16.000

As a result, one can rank the paths by their measures. The queue ofpaths in crawling precedence is as (path3, path1, path2, path4, path5).

If we assign the N as 3, the result could be further modified as thefollowings,

Log(Mea(Path1))=−7.163

Log(Mea(Path2))=−1.712

Log(Mea(Path3))=−2.064

Log(Mea(Path4))=−9.426

Log(Mea(Path5))=−12.000

Then the queue of paths becomes (path2, path3, path1, path4, path5). Theresult is more reasonable.

The method for path-based ranking of unvisited Web pages according to apreferred embodiment of the invention has been described by reference tothe figures. The system for path-based ranking of unvisited Web pagesaccording to a preferred embodiment of the invention will now bedescribed by reference to FIG. 3.

As shown in FIG. 3, the system consists of following components: anadjusting means 301 that allows users to adjust the “good-path set” ininteractive manner; a path labeling means 302 that classifies all thevisited web pages and labels each web page with the labels of a class orclasses it belongs to, the path labeling means further labels“partial-path set”; a statistic model training means 303 that learns astatistic model which generalizes the common patterns hidden in the goodpaths from the patterns of “good-path set”; and an evaluation means 304that evaluates the “partial-path sets” with the statistic model andsorts the list of the unvisited web pages with the evaluation results.In the system shown in FIG. 3, the “good-path set” and the “partial-pathset” may be stored in a storage unit.

If not otherwise stated herein, it is to be assumed that all patents,patent applications, patent publications and other publications(including web-based publications) mentioned and cited herein are herebyfully incorporated by reference herein as if set forth in their entiretyherein.

Although illustrative embodiments of the present invention have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may beaffected therein by one skilled in the art without departing from thescope or spirit of the invention.

1. A system for path-based ranking of unvisited Web pages for WWWcrawler, comprising: a storage unit which stores all paths leading tovisited web pages containing content relevant to a search as a good-pathset and all paths leading to unvisited web pages as a partial-path set;a classifier and a labeler which classifies all the visited web pagesand labels each web page with the labels of a class or classes itbelongs to; a statistic modeler which trains a statistic model forgeneralizing common patterns among all ones of the good-path set; and anevaluator which evaluates the partial-path set with the statistic modelto determine a similarity between the partial-path set and all ones ofthe good-path set and ranks the unvisited web pages with evaluationresults, wherein highest ranked unvisited web pages are unvisited webpages most likely to contain content relevant to the search.
 2. Thesystem according to claim 1, further comprising: an adjuster whichadjusts the good-path set through an interaction stage.
 3. The systemaccording to claim 1, wherein: said statistic model is described by:P(L _(i)|(L _(i-1),L _(i-2), . . . ,L ₀)), i≧0  (1) where L_(i) denotesa label of a web page and the web page is an i^(th) one in a path,wherein a path length is defined as i+1 and (L_(i-1),L-_(i-2), . . .,L₀) denotes a certain path having a length of i and whose n^(th) webpage is L_(n) where 0≦n<i.
 4. A non-signal program storage devicereadable by machine, tangibly embodying a program of instructionsexecutable by the machine to perform method steps for path-based rankingof unvisited Web pages for WWW crawling, said method steps comprising:identifying all paths beginning with a seed URL and leading to visitedweb pages containing content relevant to a search as a good-path set,and for each unvisited web page, identifying paths beginning from theseed URL leading to each unvisited web page as a partial-path set;classifying all the visited web pages and labeling each visited web pagewith labels of a class or classes each visited web page belongs to;training a statistic model for generalizing common patterns among allones of the good-path set; and evaluating partial-path sets with thestatistic model to determine a similarity between the partial-path setsand all ones of the good-path set and ranking the unvisited web pageswith evaluation results, whereby a highest ranked unvisited web pagesare unvisited web pages most likely to contain content relevant to thesearch.
 5. The non-signal program storage device according to claim 4,wherein: said statistic model is described by:P(L _(i)|(L _(i-1),L _(i-2), . . . ,L ₀)), i≧0  (1) where L_(i) denotesa label of a web page and the web page is an i^(th) one in a path,wherein a path length is defined as i+1 and (L_(i-1),L_(i-2), . . . ,L₀)denotes a certain path having a length of i and whose n^(th) web page isL_(n) where 0≦n<i.